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ABSTRACT 

Course pi a cement systems in pos t secondary education 
consist of an assessment component to estimate students' probability 
of success in standard first-year courses and an instructional 
component in which underprepared students are taught the skills and 
knowledge they need to succeed in the standard courses. Student 
success is usually defined in terms of course grades. Using a 
decision theory model to judge the effectiveness of course placement 
systems, the feasibility of eliciting students' and instructors' 
preferences for the different outcomes of course placement systems 
was studied with groups of 129 and 141 students and 9 instructors. 
The results suggest that about half of the respondents to a 
p3per-and-penc i 1 instrument provide sufficient information to develop 
coherent preferences for the outcomes of a course placement system. 
The elicited preferences differed significantly according to the 
method used (value function versus hypothetical lotteries). Responses 
of students and instructors were similar. Appendix A contains the 
questionnaire, Appendix B (two tables) discusses eliciting von 
Neumann-Morgens t e rn utilities for course grades, and Appendix C (two 
tables) discusses construct ing summary value functions for course 
placement outcomes. (Contains 4 tables, 1 figure, and 31 references.) 
(SLD) 
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Abstract 



Course placement systems in postsecondary education consist of an assessment component (to 
estimate students' probability of success in standard first-year courses), and an instructional component 
(in which underprepared students are taught the skills and knowledge they need to succeed in the 
standard courses). The effectiveness of a placement system depends on students' ultimately succeeding 
in the standard courses. Success is usually defined in terms of course grades. 

Using a decision theory model to judge the effectiveness of course placement systems, I studied the 
feasibility of eliciting students' and instructors' preferences for the different outcomes of course placement 
systems. The results suggest that about half of the respondents to a paper-and-pencil instrument provide 
sufficient information to develop coherent preferences for the outcomes of a course placement system. 
The elicited preferences differed significantly according to the method used (value function vs, 
hypothetical lotteries). Students and instructors' responses were similar. 



ERLC 



Eliciting Utility Functions for Validating Course Placement Systems^ 

Richard Sawyer 

A typical and important use of college entrance tests is course placement, i.e., matching students 
with instruction appropriate to their academic preparation. For example, students whose academic skills 
arc insufficient for them to be successful in a standard first-year English course might, on the basis of their 
test scores and other characteristics, be advised or required to enroll in a remedial English course. On the 
other hand, students with an unusually high level of academic preparation might be encouraged to enroll 
in an accelerated course or in a higher-level course. 

Most colleges i^nd universities enroll students who are not academically prepared to do work at a 
level traditionally expected of first-year students. The percentage of postsecondary institutions with some 
form of placement and remedial instruction has steadily increased in the past decade, and is now about 
(Woods, 1985; Wright and Cahalan, 1985; McNabb, 1990; "Colleges and Universities Offering 
Remedial Instruction," 1994). One suggested explanation is that American high schools have become less 
effective in preparing students for college (The National Commission on Excellence in Education, 1983; 
The Carnegie Foundation for the Advancement of Teaching, 1988; Singal, T991). Another explanation is 
that more students from disadvantaged backgrounds are attending college (Munday, 1976; College 
Entrance Examination Board, 1977; Carriuolo, 1994). 

During the past three decades, several authors have proposed using decision theory to validate 
educational selection systems. Two different general approaches are those proposed by Cronbach and 
Gleser (1965) and by Petersen and Novick (1976). Cronbach and Gleser adapted linear regression 
methodology to estimate the expected costs and benefits of using a test score or other predictor variable 
for classifying or selecting personnel. Their technique continues to be widely applied in industrial/ 
organizational settings. Petersen and Novick (1976) developed a "threshold" model based on Bayesian 
decision theory. Ben-Shakliar, Kiderman, and Beller (1994) compared these two approaches, and 
ilhistrated them using data from an admission selection problem. 

' 1 wibh to express my appreciation to Dan Anderson, Jerry Dallam, and Chuck Hinz for their help 
in collecting data f(^r this study; Mark Houst(^n for calculating the utility fimclions; and Mark Houst(M^, 
Alan Nicewander, and Julie Noble for their comments on earlier drafts of this paper. 



Sawyer (in press) proposed a statistical decision theory model for validating course placement - 
variables such as tests. The model can be used to compare the effectiveness of alternative placement 
variables in identifying underprepared students, and to determine appropriate cutoff scores on these 
placement variables. Sawyer (1994) proposed a decision theory model for measuring the effectiveness and 
worth of remedial instruction. In this paper, alternative methods are investigated for eliciting decision 
makers' preferences for course placement outcomes. 

Background 

Remedial Instruction 

At many postsecondarj' institutions, there are two levels of first-year courses: a "standard" course 
in which most students enroll, and a "remedial" course for students who are not academically prepared 
for the standard course. At some institutions, the lower-level course may be given other names, such as 
"college-preparatory," "compensatory," "developmental," or "review." Carriuolo (1994) articulated 
differences in the meanings of "remedial" and "developmental." At some institutions, there may be 
courses that require more knowledge and skills than the lowest-level remedial course, but less than the 
standard course. In this paper, only a single lower-level course is considered, and it is designated 
"remedial," to be consistent with Willingham's (1974) nomenclature. Often, remedial courses do not carry 
credit toward satisfying degree requirements. 

Though essential to placement, testing is but one component of an overall system. To be 
educationally effective, a placement system must satisfy all of the following requirements: 

1 . Students who have small chance of succeeding in the standard course (underprepared students) 
are accurately identified. 

2. Appropriate remedial instruction is provided to these underprepared students, 

3. Both the students who originally enrolled in the standard course, and the students who were 
provided remedial instruction, eventually do satisfactory work in the standard course. 
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Note that accurately identifying undcrprepared students (Requirement 1) is necessary, but not sufficient, 
for a placement system as a whole to be effective. Accurate identification is not an end, but only a 
mechanism for effectively allocating remedial instruction (Requirement 2). On the other hand, providing 
remedial instruction is itself only a means to achieve the larger goal that students succeed in college: Even 
if underprepared students are accurately identified and are provided remedial instruction, if they 
eventually drop out or fail in the standard course, then little will have been accomplished by the 
placement system. On the contrary, both the institution's and the students' resources will have been 
wasted. Van der Linden (1991) noted that a defining characteristic of course placement systems is that 
students take different treatments (courses), and the success of each treatment is measured by the same 
criterioii variable. 

One might argue that failure in the standard course can lead to positive results for students, such 
as their selecting and succeeding in another educational program better matched with their talents and 
interests. While this statement is undoubtedly true for some students, they would have done better to 
select their preferred educational programs in the first place, through appropriate counseling. This 
scenario illustrates that effective counseling is important for effective placement. This paper does not, 
however, attempt to model the effect of counseling on the outcomes of placement. 

The need for an institution to serve students who by traditional standards are academically 
unprepared for college imposes a fourth requirement on placement systenns. Even if a large proportion 
of the underprepared students are accurately identified, are provided remedial instruction, and ultimately 
succeed in the standard course, the overall result still might not be satisfactory. This would occur if an 
institulicm diverted resources to instruction in the remedial course to such an extent that the performance 
of students in the standard course was adversely affected. In other words, institutions should consider 
the tradeoffs they must make in allocating their finite resources when they provide remedial placement 
systems; such considerations may relate to institutional mission and policy, as much as to costs and to 
grades. There is ct^itroversy about the proper role o\ remedial placement in the missions of 
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postsccondary institutions. Mac Donald (1994) argued that by overexpanding its remedial programs, the 
CUNY system seriously degraded the quality of its standard-level undergraduate programs. Livel)' (1993) 
reported on efforts in different states to eliminate remedial instruction from four-year public institutions 
by designating that role to two-year colleges. 

A Decision Theory Model for Course Placement 

The decision problem can be formally defined as follows: One must select a particular decision d 
from a set D of possible decisions. A particular outcome 9 occurs, from among a set of possible outcomes 
©. A utility function n(d,Q) assigns a numerical value to the desirability of decision d when the outcome 
is 6. The exact outcome 8 that occurs is unknown to the decision maker, but there is some probabilistic 
information available about the likely values of 9. In a Bayesian decision model, this information is 
described by a subjective probability distribution on 0; the subjective probability distribution quantifies 
the decision maker's personal beliefs abc^ut the likely values of 8, given both prior beliefs and any relevant 
data collected. The Bayesian optimal strategy is to choose the decision d that maximizes the expected 
value of u(d,Q) with respect to the subjective probability distribution on 0 (Lindley, 1972). 

To apply this structure to course placement, suppose there is a cutoff score K on a placement test, 
and that: 

* test scores are obtained for all first-year students at an institution; 

* students whose test scores are less than K are provided remedial instruction before they enroll 
in the standard course, and students whose test scores are greater than or equal to K enroll 
directly in the standard course; and 

* the actual final performance in the standard course is known for all students (i.e., for students 
who are provided remedial instruction, as well as for those who are not). 

The final performance in the standard course of students who first enroll in the remedial course will, of 
C()urs(», become known later than the performance of students who enroll directly in the standard course. 
For each student, four possible events could occur, as shown in Table 1 below. 
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Table 1 

Events Associated with Identifying and Providing Remedial Instruction 
tc Underprepared Students 



Event 


Test score 


Course into which 
student is placed 


Eventual performance 
in standard course 


0) 


> K 


Standard 


Successful 


(2) 


> K 


Standard 


Unsuccessful 


(3) 


< K 


Remedial 


Unsuccessful 


(4) 


< K 


Remedial 


Successful 



Each student is classified either as being adequately prepared for the standard course (if his or her 
test score equals or exceeds the cutoff score K), or as needing remedial instruction (if the score is less than 
K). Because the classification for any student depends on K, the set of decisions (D) in this case is the set 
of possible values of K. The goal is to find the 'T^est" value of K, and to quantify the effectiveness of the 
associated instruction. 

At an institution without a placement system, the events in Table 1 could be observed as follows: 

* Randomly assign students, regardless of their test scores, either to enroll directly in the standard 
course or to enroll first in the remedial course. 

* Observe the students' eventual performance in the standard course, and note which of them 
succeed and which do not succeed. 

For each value of K, there would be a set of proportions associated with the events (1) - (4). Lei us 
suppose, temporarily, that data are collected this way; the modifications required when there is prior 
selection resulting from an existing placement system are described on p. 9. 

Let p,(K), p2(K), etc., denote the observed proportions of students corresponding to events (1), (2), 
etc., in the entire group of students when the cutoff score is K. (Then, for example, pi(K) + p4(K) is the 
proportion of students who are ultimately successful, and pjCK) f p^(K) is the proportion of students who 
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are ultimately unsuccessful. The overall usefulness of the predictions can then be evaluated in terms of 
the costs and benefits associated with each event (1) - (4). A function that assigns a value to outcomes 
such as these is called a utility function. One class of utility functions would assign different values to 
each event, and weight their sum: 

u(K;9) = w,p,(K) + W2P2(K) + w^p^iK) + W4p4(K) (1) 
where 0 < w^,...,W4 < 1. Such a function would quantify the different costs and benefits of each outcome. 
Consider, for example, the trade-offs a student must make in his or her utility. Although students pay 
tuition to take remedial courses (just as they do to take other courses), remedial courses often do not carry 
college credit. From a student^s persp>ective, the weights Wj, . . ., W4 must balance the benefit in 
performance in the standard course against the extra time and money spent on taking the remedial course. 

In principle, utility functions are person-specific, and hence need to be elicited separately for each 
student, counselor, teacher, or administrator. In practice, this is not feasible, and we must look for utility 
functions that reasonably approximate the preferences of different groups of people. 
Other models 

in the model described in Table 1, there are only two results in the standard course: "Successful" 
and "Not successful." In practice, "Successful" usually means completing the standard course with a 
particular grade (e.g., C) or higher. A more basic decision theory model, defined directly in terms of the 
grade received, would describe people's preferences more accurately. For example, instead of designating 
each student as "Successful" or "Unsuccessful" in the standard course, one could specify the student's 
completion of the course and final grade (e.g., A-F). In this case, there would be 10 outcomes (rather than 
4) in the model; such a model is described on p. 16. If we considered that some students withdraw before 
completing the standard course, then there would be 12 outcomes in the model. 

The adequacy of the model in Table 1 therefore assumes that the decision maker's preferences for 
parliciilar grades have a stop-function relationship. Petersen and Novick (1976) called such a function a 
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"threshold utility." One goal of this study was to obtain evidence about the appropriateness of threshold 
utility functions in course placement. 
Expected Utility Functions 

In practice, a utility function cannot be directly computed for the group of students for whom 
placement decisions are to be made, because the actual outcomes (students' test scores and eventual 
performance in the standard course) are not yet knowii. In (1), for example, the actual proportions p,(K), 
p.CK), etc., are not known for a particular group of students before they are tested and complete the 
standard course. These proportions must instead be estimated in some way from data on past students, 
under the assumption that future students will be similar to past students. 

The "expected utility function" is a formal mechanism for dealing with the uncertainty of outcomt»s 
in a decision theory model. It is from the expected utility function that decisions on the effectiveness of 
a placement system can be made. In Bayesian models, an "expected utility function" is the average 
(expected) value of a utility function n(d,d) with respect to the decision maker's subjective probability 
distribution for the outcomes 0. In the example previously given, 

u' (K) = Eel u(K,e) 1= w,p,(K) + W2p2(K) + w,p,{K) + W4p4(K) (2) 
where pj(K) = EqI p,(K) ], p2(K) = EqI p2(K) ], etc., are estimated from a past group of students. In the 
Bayesian model, the estimates pi(K), p2(K), etc., are the expected values of the corresponding observed 
proportions with respect to the decision maker's subjective probability distribution for students' test scores 
and C(^urse grades. In the terminology of Bayesian statistical inference, the subjective probability 
distribution for test scores and course grades is specified by a "predictive density" for their joint 
distribution. The predictive density is based on prior beliefs about the joint distribution and on data 
obtained from a particular group of past students. Although simple in concept, Bayesian statistical 
methods can be mathematically formidable in real applications. When prior beliefs arc vague or as saniple 
sizes become large*, however, Bayesian estimates are, for practical purposes, similar to much simpler 
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estimates based on classical sampling theory (i.e., estimates based only on an assumed model and on data; 
DeGroot, 1970). 

Sawyer (in press) described a simple procedure, based on sampling theory, for estimating the cell 
probabilities pi(K), p2(K), etc. The first step is to estimate the relationship between success in the standard 
course and a placement test score using a logistic regression function: 

P[Y=1 |X=xl = ( 1 + e-"-P'' (3) 

where Y = 1, if a student is successful, 

= 0, if a student is unsuccessful; 
and X is the student's score on a placement test or other placement variable. The numbers a and p in 
Equation (3) are unknown parameters that are estimated from data on the test scores and on the 
success/failure variable Y for a group of enrolled students. The regression function Ps(x) of students who 
enroll directly in the standard course and the regression function Pr(x) of students who enroll first in the 
remedial course are estimated separately. 

Once estimates a and h have been obtained for the unknown parameters a and p, the conditional 

probabilities of success %M and Pr(x) can be estimated by substituting q and b in Equation (3). From 

the estimated conditional probabilities, the proportions for the four events described in Table 1 can be 
easily calculated. For example, the proportion of students associated with Event (1) in Table 1 can be 
estimated by: 

p,(K)= E^s(x)*n(.v) / N (4) 

where = estimated P jV = 1 | X = xl for students who enrolled directly in the standard course, 

K = the minimum score required for enrollment in the standard course (cutoff score), 

n(x) ^- the number of students in the placement group whose test score is equal to x, and 

N E n(x), the total number of students in the placement group. 
The proportions for Events (2), (3), and (4) can be estimated similarly. 

erJc « 1 
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Note that the summations in Equation (4) are based on the x-values (e.g., test scores) of all the 
students in the placement group (the set of sludents for whom placement decisions are made), not just 
the students who complete the course. In practical terms, the placement group will usually consist of all 
first-time entering students with test scores, regardless of which course they actually enroll in. Of course, 
one could also define a placement group for students in a particular program of study (e.g., business) or 
with particular background characteristics (e.g., minority students). 

At an institution with an operational placement system with cutoff score K^, we can estimate Ps(x) 
only from data with x > K<v and we can estimate Pr(x) only from data with x < K^. The reason is that 
students whose test scores are below the cutoff score Ko do not enroll directly in the standard course, and 
therefore do not have performance data unaffected by remedial instruction. Sawyer (1993) noted, 
however, that the logistic regression model (3) can be conveniently extrapolated to test scores below the 
current cutoff score K^. Schiel & Noble (1993) compared logistic regression functions estimated from 
truncated subsets of a data set that was not subject to prior selection. They found that when the 
truncation involved less than 157r. of the po^^ulation, the resulting errors were small, but that large 
amounts of truncation (e.g., 50%) resulted in large errors. Houston (1993) did computer simulations to 
examine the effects of truncation on the accuracy of estimated conditional probabilities of success. He 
found increases in standard error of 6%, 30%, and 43% when the placement group was truncated at the 
25th, 5()th, and 75th percentiles, respectively, as compared to the stanoard error associated with no 
truncation. 
Optimal cutoff scores 

If the expected utility i/(K) = EoIu(K;9)l attains a maximum value at some cutoff score Ko, then using 
Ko as a cutoff score will result in a greater expected utility for the group than using any other cutoff score. 
Furthermore, if K,j is between the minimum and maximum possible scores on the test or other placement 
variable, then the effectiveness of the placement system as a whole is supported. On the other hand, if 
u' is an ini reasing function, then ihQ effectiveness of the placement variable is called into question — the 
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placement variable is not able to discriminate between students who should enroll directly in the standard 
course and those who should first take the remedial course. Finally, if u' is a decreasing function, then 
the effectiveness of both tlie placement variable and the remedial course is called into question. Of course, 
all of these inferences depend on the validity of the success criterion variable. 

Eliciting Utility Functions 

if the decision model and optimal cutoff score are to be useful in real applications, the utility 
function must accurately describe the preferences of the decision makers. In the model described by Table 
1, for example, we need some way to quantify students' and instructors' preferences for success in the 
standard course, as balanced against the extra time and cost associated with taking the remedial course. 

There is a vast literature on eliciting (i.e., assessing) utility functions. One important characteristic 
distinguishing various utility theories is whether they are deterministic or stochastic: 

* A value function measures the satisfaction of any sort of "want" without regard to uncertainty. 
For example, some economists model the satisfaction that an individual receives from consuming 
commodities. The key characteristic of a value function is that it assigns numerical values to the 
subjective worth of outcomes without regard to uncertainty (Yates, 1990). 

A simple example of eliciting a value function would be to ask an individual to rank each 
possible outcome on the following Likcrt scale: 

l="dislike very much", 2="dislike", 3='*dislike a little", 7="like very much") 
Note that in this example, the assignment of values to outcomes is done outside any context of 
uncertiinty or risk. 

* A von Neumann- tviorgenstern utility, in contrast, is explicitly defined in terms of uncertainty. The 
standard assumption in von Neumann-Morgenstern (abbreviated hereafter as vN-M) theories 
is that the decision maker has a preference relation < over the set n of probability distributions 
on the outc(U-ne space 0 (rather than on 0 itself), and that -< satisfies an appropriate set of 
axioms (e.g., transitivity). Then it can be shown that there exists a real function u on 0, such 
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that for distributions p, q 6 n, p < q if, and only if, E^lu] < E^fw]. The function u is unique up 
to positive, linear transformations; therefore, one can without loss of generality assign the value 
0 to the least favorable outcome and the value 1 to the most favorable outcome. Note that vN-M 
utility functions are defined in terms of probability; therefore, their elicitation is naturally done 
in reference to hypothesized probability distributions. See Farquhar (1984) for a comprehensive 
reviev^ of different strategies for eliciting vN-M utility functions. 
The principal advantage of value functions is that they are easy to elicit, because they do not require any 
reference to uncertainty or risk. Tlie principal advantage claimed for vN-M utility functions is that they 
are more realistic, because they reflect the decision maker's fillings about both tlie inherent v^orth of the 
outcomes, and the risk involved in making choices. (On the other hand, this realism is elicited in the 
context of hypothetical situations!) Although both value functions and vN-M utility functions can 
formally be used in expected utility models (Yates, 1990), they are not the same, and can lead to different 
decisions. I shall follov^ Yates' convention in reserving the term "utility" to refer specifically to a vN-M 
utility function, and the term "value function" for a function that does not consider risk. 

One class of methods for eliciting vN-M utility functions is called "probability equivalence" methods. 
Probability equivalence involv(»s asking a decision maker to determine the probability p for which he or 
she is indifferent to obtaining Outcome 0^ with certainty, and a gamble involving Outcome 6, with 
probability p and Outcome B, with probability 1-p. Farquhar (1984) denotes this relationship as 

~ I ^1/ P/ ^1 I- Different probability equivalence methods involve different ways of choosing the 
outcomes in the hypothetical gambles. Novick and Lindley (1979), for example, order the n outcomes 
0 = ;/(9o) < u(d]) < . . . < t/(0n.,) < w(6n) = 1. They then make comparisons involving the n-1 adjacent 
outcomes: 9, ~ [B,,i, p„ G,.J. Finally, they solve the resulting system of linear equations: 
1/(9^) = p, * w(9,,i) + (1-p,) * iH9,.i)/ where i=l,...n"l. Novick and Lindley also consider additional 
gambles involving more distant comparisons, such as 9, - I0,.2, r„ 9,2), to check the consistency (also 
called "coherence") of the elicited utilities. 
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Decision theory provides an intellectually attractive method for studying the effectiveness of 
remedial instruction. Its practical feasibility in this application, hov^ever, needs to be proven. Among the 
feasibility issues, eliciting the preferences of students and instructors is certainly crucial: If these decision 
makers are unable to provide information that accurately reflects their preferences and is inexpensive to 
collect, then the method will be only a toy of statisticians, rather than a practical means for improving 
postsecondar^' education. 

The purpose of this study v^as to obtain preliminary answers to the following questions: 

1. Is it feasible to elicit utilities by a paper-and-pencil questionnaire? 

2. How do different analytic schemes for eliciting utilities affect the results? 

3. Is the threshold utility a reasonable approximation to students' or instructors' utilities? 

4. How do the utilities of students differ from those of instructors? 
The reasons for posing these questions are discussed below. 

Question 1 has implications for the feasibility of eliciting utilities in a routine and large-scale 
settings. Sophisticated interactive computer systems (e.g., Isaacs & Novick, 1978) have been developed 
for eliciting utilities; these systems have internal mechanisms for detecting and correcting inconsistencies 
in decision makers' responses, either by asking for additional information, or by smoothing, or both. It 
would certainly be more economically and practically feasible, however, for institutions to administer 
paper-and-pencil questionnaires than to maintain or subscribe to an interactive computer system. 

Question 2 also has implications for eliciting utilities in large-scale settings. Value functions are 
much easier to elicit than vN-M utility functions. If the elicited value functions of most decision makers 
closely approximated their elicited vN-M functions, then one would need to elicit only the value functions. 

To answer Question 3, let u{G) denote an instructor's value function or vN-M utility for grade G, 
and let denote "much less than." If we observed the following result: 

iKF) < u{D) « uiC) < //(B) < j/(A), M 



thcMi a threshold utility with respect to the C-or-higher success criterion would be supported. If we found 
that 

iKF) < uiD) < uiO « u{B) < uiAl (7) 
then a threshold utility with respect to the B-or-higher success criterion would be supported. 

Because students and instructors obviously have different roles in education, they may well have 
different preferences (Question 4). Two other groups, counselors and administrators, are also important 
decision makers in course placement systems, and could have utilities that differ in important ways from 
those of both students and instructors. Unfortunately, it was not possible in this study to administer a 
questionnaire to counselors and administrators. Future studies will include them. 

Definitive answers to these questions undoubtedly depend on many educational and background 
variables, and could be the goal of an entire research agenda. For example, utilities of students and 
instructors at different types of institutions (e.g., 4-year liberal arts colleges, state universities) may differ 
from those of students and instructors at community colleges. This study, it is hoped, provides initial 
"order-of-magnitude" results, as well as guidance on how to design more sensitive studies in the future. 

Data 

1 administered questionnaires to the following groups of people: 

* Croup 1: Students who enrolled in first-year remedial or standard English or mathematics 
courses at a community college in the midwest (n==129). 

* Group 2: Students who enrolled in basic algebra or calculus at a public university in the 
midwest (n=l41). 

* Group 3: Tlie instructors of the students in Group 2 (n=9). This group included 1 faculty 
member and 8 graduate students. 

To make the questionnaire it(»ms more meaningful to the respondents, I developed a separate 
questionnaire for each group. Tlie questionnaires are reproduced in Appendix A. (The two institutions 
are given the* fictitious names "Midwest Community College" and "Midwest Public University,") 
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Backgrouud Information 

Part 1 of each questionnaire asked about respondents' course taking (or course teaching) experience, 
as well as background information. These questions will be used in future studies to deternnine whether 
particular groups of respondents have particular difficulties in providing preference information, and 
whether their preferences differ substantially from each other. 
Value Function 

Part 2 of each questionnaire elicited a value function for the grades of B, C ^^nd D in the standard 
course. To simplify the respondents' deliberations, this question ignored the possibility that a student 
might withdraw (W) or obtain an incomplete (I) in the standard course. Respondents were presented with 
a scale ranging from 0 to 100, and incremented in units of 10. In the student version of the questionnaires, 
the scale was intended to measure satisfaction with particular grades, with F indicating a satisfaction of 
0, and A indicating a satisfaction of 100. In the instructor version of the questionnaires, the scale was 
intended to measure satisfaction with student academic performance levels associated with different 
grades. Respondents were asked to mark the letters D, C, and B over the points on the scale that reflected 
their satisfaction with these grades.^ This method of eliciting value functions is called "Stevens' 
magnitude estimation with modulus" (Falmagne, 1985). 
vN-M Utility 

Part 3 of the student questionnaire elicited vN-M utility functions for the grades B, C, and D. 1 
asked students to consider either earning a particular grade G for sure, or else entering a lottery involving 
grades Go and G,, where Go < G < Gj. In the lottery, students would earn grade Go with probability p, 
or grade G, and probability 7 -p. Here is an example item: 



^ Respondents were warnc»d not to confuse the "siti^faction sca1(^" with the "percent correct scales" 
sometimes used to assign grades (e.g., A=.90-1(X)). 
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Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either a C or an F. 

How large would the chances of earning a C have to bo before you would prefer taking a chance [Option (2)\ 
to the sure grade of D (Option (1)]? 

I would want the chances of earning a C to be % before taking a chance. 

Students were given all possible logical combinations (n=10) of G, Gq and G,. In an earlier pilot study 
(Sawyer, 1994), the lotteries were also presented as choice tasks. In the choice tasks, several values of p 
were displayed in a table, and students were asked to mark the value of p for which they were indifferent 
between the sure grade and the lottery. I found that students responded no better to the choice tasks than 
to the direct elicitation items, and that the resulting utility functions wero similar. 

Part 3 of the instructor questionnaires also elicited vN-M utilities. The only difference between Part 
3 of the instructor questionnaires and Part 3 of the student questionnaire is that the lotteries in the 
instructor questionnaire were stated in terms of grade distributions, rather than probabilities. For 
example: 

Suppose I asked you tu choose between the foIlo\sing two scenarios in your course: 

(1) A uniform result, in which all students perform at D level, or 

(2) A mixed result, in which P percent of the students perform at C level, and all the rest perform at F level. 

I low large would the percentage P of students who perform at C level have to be before you would prefer the 
mixed result j Scenario (2)1 to the uniform performance at D level | Scenario (V\7 

\ would want P = before selecting the mixed result. 

Of course, the phenomenon described by the second item is the expected result of the phenomenon 
described by first item, but the two items are not, strictly speaking, asking the same question. 1 elected 
t(^ use the second style, because it is more natural from the perspective of instructors.^ 

To reduce potential order effects in eliciting vN-M utilities, 1 created two forms of the student and 
instructor questionnaires. The two forms differt^d only in the order of the lotteries. For example. Item 
1 of Form A pertained to the comparison D - ( C, p, F | and item 10 pertained to the comparison 



^ Instructors nevertheless objected to the hypothetical lotteries, because of their artificial qualit>'. 
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B ~ 1 A, C |. In Form B, the items were presented in reverse order. 1 then "spiralled" the two forms 
(i.e., collated them before distributing them). 
Rerfiedial Instruction 

Part 4 of the questionnaire considered the effectiveness of remedial instructior*. If grades A-F in 
the standard course are assumed to define the final results of a student's involvement with a course 
placement system, then the outcome space consists of 10 elements: 

(-) =( (S,A>, (K,A), (S,B), (R,B), (S,C), (R,C), (S,D), (R,D), (S,F), (R,F) L 
where S denotes taking the standard course directly, R denotes taking tlie remedial course before taking 
the standard course, and A, B, C, D, F are the grades a student eventually earns in the standard course. 
Eliciting vN-M utilities for all these outcomes seemed, on its face, to be infeasible in a paper-and-pencil 
format. Even eliciting a value function for 10 outcomes seemed impractical. Therefore, 1 elected to elicit 
a value funcH(<n for the outcomes associated with taking the remedial course, relative to the outcomes 
asbocUie J with taking the standard course directly. The values associated with taking the standard course 
directly were taken to be those elicited in Part 2 of the instrument. 
Adttiiiiistratiou 

The student questionnaires were distributed by instructors at the end of one class, and then 
collected at the beginning of subsequent classes. This method of administration is obviously vulnerable 
to self-selection effects; unfortunately, the instructors were not willing to give away instructional time for 
students to complete tiieir questionnaires in class. Given the choice between potentially biased data, and 
no data at all, 1 acceded to the conditions demanded by the instructors. 

The instructors completed their questionnaires in the same manner. 

Analysis 

Ft)r both the student data and tiie instructor data, 1 computed coherence indicators for the value 
functions and coherence rates for the vN-M utility function. 1 summarized the distribution of the 
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cohcrcncc indicators and rates, the value functions, and the vN-M utilities over the total group of 
respondents and over respondent subgroups. 
Value Fimctim Cohcrmce Indicator 

I computed a coherence indicator (denoted "CHRINDl") for the responses to Part 2. For an elicited 
grade value function gof, CHRIND1 = 1 if 0 < gvfiD) < gvfiO < gvf(B) < 100, and CHR1ND1=0 otherwise. 
Calculatbi^ vN-M Utilities aJid Coherence Rates 

The data from each comparison in Part 3 can be represented by a linear equation. For example, the 
data from the comparison C ~ [B, p, Dj can be represented by the linear equation 
u{C) = p * //(B) + (1 - p) * u(D). Because there were 10 comparisons in Part 3, and because there are three 
"unknowns" (i/(B), iKC), and u{D)), the responses to Part 3 could result in a maximum of 10 linear 
equations in 3 unknowns. It can be shown that of the resulting 120 = (^3) systems of 3 linear equations 
in 3 unknowns, only 108 are of full rank. (Appendix B contains a listing of the 108 full-rank systems.) 
Therefore, each respondent could, in principle, have 108 different solutions for 2/(B), w(C), and w(D). 

In practice, of course, a respondent might not provide equivalence probabilities p for all 10 
comparisons, and so might have fewer than 108 sets of elicited utilities. For each respondent, all the 
systems of linear equations for which there were valid data were solved. Some of these resulting solutions 
were "coherent" in the sense that 0 < w(D) < u(C) < u{B) < i. Incoherent solutions resulted from 
inconsistent responses to the items (for example if a respondent reported probabilities p^ < p2 in the 
comparisons B ~ (A, Fj and C ~ I A, p^, FI). The "coherence rate" for a respondent was defined as the 
number of coherent solutions divided by 108. The coherence rate is an indicator of how well utilities can 
be elicited from a respondent. Note that the coherence rate (CR) pertains only to the vN-M utility, while 
the coherence indicai " pertains to the value function. For each respondent, I calculated the CR and the 
mean of the utility values i/(B), z/(C), and z/(D) associated with coherent solutions. 
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Course Placement Outcomes 

For each respondent, I computed a summary value function for course placement outcomes. Tlie 
summary value function measures a respondent's preferences for taking the remedial course or not, 
relative to different grades in the standard course. I computed the summary value function by combining 
the information elicited from Part 4 of the questionnaire v^ith the grade value function information elicited 
fron^ Part 2. For example, as before, let (R,A) denote the outcome that a student takes the remedial course 
before taking the standard course and earns an A; let (S,B) denote the outcome: that a student takes the 
standard course directly and earns a B; and suppose that a respondent's choices in Part 4 of the 
questionnaire indicate that (R,A) y (S,B). Th«^n, a summary value function svf can be imputed by 
interpolating bctv^een ^=gvf{A) and gvfiB): svf(RA)=-50 + .50*^/(8). 

It is possible to do such imputation consistently provided the respondent's choices in Part 4 are 
coherent (consistent). For exarnple, the follov^ing tv^o choices are incoherent: 

a. (R,A) y (S,B) 

b. (S,C) y (R,A). 

Of the 2'° = 1024 possible sequences of choices, only 14 are coherent. For each respondent, 1 computed 
an indicator CHCSEQ: CHCSEQ=1, if the respondent's choice sequence v^as coherent; and CHCSEQ=0, 
othenA^ise. See Appendix C for details on the coherent choice sequences. 

The respondi^nts for v^hom a coherent summary value function can be computed are those for 
whom CHRIND1=1 and CHCSEQ=1. 1 therefore computed a coherence indicator for the summary value 
function CHR1ND2=CHCSEQ*CHR1ND1. 
Summary Statistics 

Each respondent had a grade value function with its coherence indicator CHRINDI; mean elicited 
vN-M utilities with their associated coherence rate CR; and a summary value function with its associated 
coherence indicator CHRIND2. 1 summarized the distribution of these statistics over all respondents and 
over the respondents in each Group (as defined on pp. 13-14), using the minimum, median, and maximum 
value of each statistic. 

2i 
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Rcsults 

Response Rate 

Of the 191 community college students in Group 1 who were given the opportunity to respond to 
the questionnaire, 129 (677o) did so. Of the 320 university students in Group 2 who were given the 
opportunity to respond, 141 (447o) did so. All 9 of the mathematics instructors in Group 3 responded to 
the questionnaire. 

The response rates for both student groups (particularly, for Group 2) leave open the possibility for 
self-selection bias. In other words, the results obtained from the sample collected might have been 
affected by the characteristics of the students who were inclined to respond to questionnaires. 

The sample of mathematics instructors, while not influenced by self-selection biases, was very small. 
The results for the instructors should therefore be interpreted with caution. 
Cohereticc 

Table 2 on the following page summarizes the distribution, by Group, of the coherence indicators 
(CHRINDl and CHR1ND2), and of the coherence rate (CR). About 2/3 of the overall group provided 
coherent grade value functions (CHRINDl). About 1/5 of the incoherent grade value functions were 
classified as incoherent because the respondents stipulated that ^^/(D)=:0; if this particular response had 
been classified as coherent, then about 3/4 of the respondents would have provided coherent grade value 
functions. Nearly all of the remaining incoherent responses were due to incomplete data (e.g., respondent 
provided f^f{B), and gvfiO, but not gvf(D)). The public university mathematics students responded 
coherently more often (75%) than the community college students (577o). 

The median coherence rate for the vN-M utility function was .35. Although the maximum observed 
CR was .90, about 6% of the total group had a zero CR; and only ab<^ut 20% of the respondents had a 
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CR > .50 (not shown in Table 2). The public university mathematics students again performed better (median CR=.44) 
than the community college students (median CR=.35). 

About 787r of the respondents supplied coherent sequences of choices in Part 4. A coherent summary value function 
could be constructed for about 46% of respondents, 
Elicited Utilities 

Table 3 on the following page shows the distributions, for the different respondent groups, of the elicited grade 
value function and the vN-M utility function. Figure 1 on the page following Table 3 pictorially displays these results 
for the total group of respondents. Both Table 3 and Figure 1 are based on the 188 responses for which CHRINDl > 0 
(grade value function) or the 262 responses for which CR > 0 (vN-M utility). 

Total ,^roup. The most apparent result for the total group is that '.he median vN-M utility was significantly higher 
than the median grade value function, particularly for the grades C and D. For example, the median grade value function 
for C was ,50, and the median vN-M utility function for C was .75. This result is also true of all three respondent groups. 
This result is consistent with one obtained in an earlier pilot study (Sawyer, 1994). 

The vN-M utility was also more variable over respondents than the grade value function. For example, the elicited 
vN-M utility for the grade C ranged from .01 to .98; but the grade value function for C ranged from .10 to .85. This result 
was also true of all three respondent groups. 

Suspecting that these two results might be related to the quality of individuals' responses to the hypothetical 
lotteries in Part 3 of the questionnaire, 1 studied the relationship between the difference gvf{G) - vNM{G) and the 
coherence rate CR(G), for the grades G = B, C, and D. 1 found no relationship between these variables for any respondent 
group. Moreover, there was no relationship between these variables in any spiralled form of the questionnaire. 

Group comparisons. The instructors' median grade value function was very similar to that of the students. Tlie 
instructors' median vN-M utility was about .10 lower than that of the students. 1 found a similar result in the earlier pilot 
study (Sawyer, 1994). 
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Sximmary Value Function 

Recall that we are assuming that the outcomes of a course placement system are elements of the set 
e = { (S,A), (R,A), (S,F), (R,F) }. Table 4 shows the distribution, over the total group of respondents, of the imputed 
summary value function CHRIND2 for this set. According to the medians in Table 4, the typical respondent would be 
as satisfied with enrolling in the standard course directly and earning a Q as he or she would with first taking the 
remedial course and earning an A or B in the standard course. Taking the remedial course and earning a C in the 
standard course was much less desirable; and taking the remedial course and earning a D in the standard course was 
valued hardly more that getting an F. The results for the instructors were similar to those of the students. 

I also computed a hybrid "summary value function" by combining the elicited vN-M utility from Part 3 with the 
choice sequence in Part 4. i obtained result like those in Table 4, but shifted to higher numerical values. 

Table 4. 

Distribution of Summary Value Function 
for Course Placement Outcomes 
(Total Group) 



Course placement 
outcome 


Summary value function 


Min. 1 


Med. 1 Max. 


(S,A) 


1.00 


1.00 1 1.00 


(R,A) 


.18 


.54 1 .91 


(S,B) 


.40 


.75 1 .95 


(R,B), 


.09 


.55 1 .88 


(S,C) 


.10 


.50 I .85 


(R.C) 


.08 


.28 1 .78 


(S,D) 


.05 


.10 1 .75 


(R,D) 


.03 


.05 1 .38 


(S,F) 


.01 


.03 1 .19 


(R,F) 


.(X) 


! .00 1 .(X) 



Ndlf: Stntisticii arc based on 152 case? with a cohorotit grade 
value function and a coherent choice sequence. 
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Discussion 

As postscxrondary institutions spend more resources on placing their first-year students into appropriate courses, 
they will be challenged to document the effectiveness of their placement systems. Evaluating complex systems requires 
presenting evidence on multiple measures. In course placement, for example, pne could document per-student costs of 
testing and remedial instruction; survey students, faculty, and staff on their satisfaction v^ith various components of the 
system; present statistics on success rates; etc. Decision theory provides another way to develop an indicator of program 
effectiveness: If the expected utility associated with the cutoff score on a placement variable significantly excc^*ds the 
expect€»d utility associated with the minimum possible score, then one has evidence that the placement system is accruing 
a net benefit to its users. 

About half of the university students and about two-thirds of the community college students suPv'eyed completed 
the questionnaire. Of those who completed the questionnaire, about 2/3 provided enough information to develop a 
coherent grade value function, and slightly less than half provided enough information to develop a coherent summary 
value function. Although 947t of the respondents had a positive,'vN-M coherence rate, the typical respondent provided 
enough information to elicit only 357v of the possible vN-M utility functions. These results suggest that institutions can 
not realistically expect to elicit utilities by routinely administering pa per-and -pencil questionnaires to their instructors 
and students (see Question 1, p. 16). Institutions would instead need to provide special instruction and motivation to elicit 
valid utilities for most students and instructors. An interactive computer elicitation program might increase validity by 
giving respondents an opportunity to corrc^ct inconsistencies. The difficulty and expense of implementing such a computer 
program, however, would seem to make it unattractive to institutions, even today. 

A more realistic strategy would be to elicit utilities for different groups of people at a variety of institutions, and 
attempt to make some kind of generalizations. For example, we might find that the summary value functions of students 
in community colleges fall into two general clusters, say for "risk-taking" and "risk-averse" students. Other community 
colleges could apply one or both of these summary value functions in developing their expected utility indicators. 

The results of this study also suggest that there is a pronounced methodological effect on elicited utilities. The vN- 
M utility function was systematically higher than the grade value function (Question 2). Moreover, this difference 
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transcended differences on any other variables 1 investigated. One possible explanation for this result is that hypothetical 
lotteries bring out risk aversion in people: people v^ill, for example, demand a very probability of an A before trading 
a certain grade of B for a lottery in which they might earn an A or an F. The method one prefers depends partly on one's 
philosophical orientation and partly on the intended use of the model: An orthodox Bayesian decision theorist v^ould 
deny that the concept of utility has any meaning outside the context of probability, v^hereas most of the respondents in 
this study complained about the artificial quality of the vN-M lotteries. If the major purpose for eliciting a preference 
function is to develop v^eights for an indicator of overall program effectiveness that v^ould be reported along v^ith more 
specific indicators, then there v^ould seem to be little accuracy lost (and maybe some to be gained) in using a value 
function. 

Figure 1 suggests that neither the grade value function nor the vN-M utility function are well approximated by step 
functions. Therefore, the threshold utility is not a very accurate description of most people's preferences (Question 3). 
This result is a pity, because the threshold model is much simpler to work with mathematically and to explain! 

Within the limitations of the data in this study, neither the grade value functions nor the summary value functions 
of students differ significantly from those of instructors. This conclusion should be considered more tentative than the 
others, given the small number of instructors. 

Conclusions 

A college course placement system consists of an assessment component and an instructional component. The 
effectiveness of the system as a whole depends on both components. Statistical decision theory can be used to describe 
the possible outcomes of course placement systems. By eliciting a preference function of the outcomes, and by averaging 
the function with respect to a probability distribution, one can evaluate the effectiveness of the course placement system 
and select optimal cutoff scores. Preference functions may be categorized according to whether they are deterministic 
(value functions) or stochastic (vN-M utility functions). 

In a study at a midwestem community college and public university, about 2/3 of the respondents were able to 
supply coherent grade value functions, and slightly less than half were able to supply coherent summary value functions 
for course* placement outcomes. Performance on hypothetical lotteries used to elicit vN-M utilities varied significantly 
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among individuals, but most respondents were able to supply enough information to calculate at least one utility function. 
The vN-M utility values v^ere typically much larger than the grade function values. The median results for the instructors 
and students v/eve very similar. 
Future research 

In fall 1995 I hope to elicit preference functions for large samples of students, instructors, and support staff at several 
institutions. 1 will revise the questionnaire to elicit only value functions 1 will attempt to obtain duplicate measurements 
for many of the respondents, so that I can estimate reliabilities. 

I also hope to collect dati on the placement variables and course grades of the students. By combining these two 
data sets, I will obtain evidence about the perceived effectiveness of the placement systems at these institutions. 
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Form B 



Students' Satisfaction With Grades 
and Course Placement Decisions 



A research project by Richard Sawyer, ACT, 
with the cooperation of Midwest Community College 

Fall, 1994 



Purpose of this Study 

There are benefits, risks, and costs associated with the decision to go to college. Part of ACT's 
work involves helping students decide which courses to take. I want to learn about the things 
you think about in making decisions about your courses. 

! will ask you some questions about your academic work at Nlidwest, and about your 
preferences for different grades and course placement decisions. This questioni.. e is not a test 
— there are no right or wrong answers. I do not ask you for your name or other identifying 
information on the questionnaire, so your answers will be anonymous. I have written a 
number on the top of this page, but only to help me keep track of which questionnaires are 
given to which classes. 

Your instructor will distribute this questiorjnaire in class. Please take it home, answer the 
questions, and then bring it back to the next class meeting, where your instructor wih collect 
it 

There is a chance that you will receive a questionnaire from two different instructors. If you 
do receive two questionnaires, then complete only the first one you receive. At the top of the 
second questionnaire, just write "Second" and return it to the instructor at the next class 
meeting. 

The questions are grouped into four parts. As soon as you finish one part, please continue on 
directly to the next part. The entire questionnaire should take 15-30 minutes to complete. 

The information you give will be used to enhance the services ACT provides to students in the 
future. I sincerely appreciate your cooperation. 

erJc i 



Part 1 

Background Information 



1. Please check (/) the appropriate boxes to indicate whether you have taken, or are currently 
enrolled in, any of the courses in the table below. Also indicate eith3r the grade you 
received in the course (if you have already taken it), or the grade you expect to receive (if 
you are currently enrolled in the course). 



Course 


Check here it j 
you have 1 
already taken I 


\jraae 

you 
received 


Check here if you i 
are currently 1 
enrolled 1 


Grade you 
expect to 
receive 


PA Reading 










PA Writing 










Elements of Writing 










Composition I 










Composition II 










College Writing 










PA Mathematics 










Beginning Algebra 










Intermediate Algebra 










Finite Mathematics 










Pre Calculus 










Mathematics for Dec. Making 










Statistical Ideas 










Other mathematics courses 
(please specify): 











What general program or major are you enrolled in at Midwest? 



When did you first start taking courses at Midwest? 
(month and year) 

When did you first enroll in your current program at Midwest? 
. (month and year) 

What is your gender? 

Female 

Male 

What is your age? 
years 

Which of the foUowmg statements best describes your goals about the grades you earn 
courses at Midwest? (Check one only.) 

I don't mind earning a few Ds, so long as I receive credit for all my courses. 

It is important for me to earn only As, Bs, or Cs in my courses. 

It is important for me to earn only As or Bs in my courses. 

It is important for me to earn all As in my courses. 
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Part 2 
Course Grades 



Students want to earn as high a grade in a course as they can. Naturally, everyone would be more 
satisfied with an A than with a B, or with a B than with a C, and so forth ~ but what about your 
relative satisfaction? Would you, for example, feel twice as satisfied with an A as with a B? 

I want to find out your relative satisfaction with grades in the standard courses you take. (A 
"standard course" is a for-credit course that you need to pass to satisfy the requirements of your 
program at Midwest.) In answering the questions, please think of a standard course that is typical 
of those you are taking or have taken. 

The line below is meant to suggest your relative satisfaction with the different letter grades. The letter 
grade of F is associated with 0% satisfaction, and the letter grade of A is associated with 100% 
satisfaction: 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Please indicate on tliis line your relative satisfaction with the grades of B, C, and D by writing them 
above an appropriate point on the line. For example, if you would be about half as satisfied with a 
B as with an A, then you would write a "B" above the 50% mark. 

NOTE: Your responses should reflect your satisfaction with particular grades in a standard course. 
Your responses do not have to correspond to, a percent-correct grading scale (for example, where the 
grade A represents 90% or more correct). 
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Parts 
Course Grades (conf d) 



4 



Sometimes we are given a choice between receiving a certain prize for sure, or else taking a chance 
on winning a better prize. For example, a contestant on a television game show may be given the 
choice either of winning a fancy color television for sure, or else having a 50% chance at winning 
$2,000 cash (and a 50% chance of winning nothing). If the contestant already has a color television, 
then he or she might be willing to take a 50% chance at winning $2,000 cash. On the other hand, 
the contestant might choose the certain prize of the color television if he or she doesn't have one. 

The following ten questions ask about your satisfaction with different grades in this way. Each 
question asks you to think about either earning a certain grade for sure [Option (1)1, or else taking 
a chance, in which you might earn a higher grade, but also might earn a lower grade [Option (2)1. 

In answering these questions, please think of any course that you need to pass to satisfy the 
requirements of your program at Midwest. 



1. Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either a C or an F. 

How large would the chances of earning- a C have to be before you would prefer taking a chance' 
[Option (2)] to the sure grade of D [Option 

I would want the chances of earning a C to be % before taking a chance. 



Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either a B or an F. 

How large would the chances of earning a B have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of D [Option (V]l 

I would want the chances of earning a B to be % before taking a chance. 
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Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either an A or an F. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of D [Option (!}]? 

I would want the chances of earning an A to be % before taking a chance. 



Suppose 1 offered you the choice of: 

(1) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either a B or an F. 

How large would the chances of earning a B have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of C [Option fl)]? 

I would want the chances of earning a 6 to be % before taking a chance. 



Suppose I offered you the choice of: 

(!) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either a B or a D. 

How lar"'^ would the chances of earning a B have to be before you would prefer taking a chance 
[Option v2j] to the sure grade of C [Option 

I would want the chances of earning a 6 to be % before taking a chance. 



Suppose I offered you the choice of: 

(1) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either an A or an F. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of C [Option (!}]? 

I would want the chances of earning an A to be % before taking a chance. 
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7. Suppose I offered you the choice of: 

(1) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either an A or a D, 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of C [Option (1)]? 

I would want the chances of earning an A to be % before taking a chance. 



8. Suppose I offered you the choice of: 

(1) Earning exactly a grade of B for sure, or 

(2) Taking a chance, where you could earn either an A or an F. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of B [Option fl )]? 

I would want the chances of earning an A to be % before taking a chance. 



9. Suppose I offered you the choice of: 

(1) Earning exactly a grade of B for sure, or 

(2) Taking a chance, where you could earn either an A or a D. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of B [Option H)]? 

I would want the chances of earning an A to be % before taking a chance. 



10. Suppose I offered you the choice of: 

(!) Earning exactly a grade of B for sure, or 

(2) Taking a chance, where you could earn either an A or a C. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of B [Option (1)]? 

I would want the chances of earning an A to be % before taking a chance. 
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Part 4 
Course Placement 



Let a "standard course" be a for-credit course that is required for your program. For 
example, many entering students may need to pass Composition I to satisfy the 
requirements of their programs at Midwest. 

One purpose of a course placement system is to determine whether a student is ready 
to take a particular standard course. If a student is not ready to take the standard 
course, he or she can instead enroll in a "developmental course" to acquire the skills 
needed to succeed in the standard course. At Midwest, for example. Elements of Writing 
would be considered the developmental course for Composition /. 

Taking a developmental course will tend to increase a student's chances of success in the 
standard course. However, taking a developmental course also has disadvantages—it 
increases the time required to complete your program, and it costs additional money. 
Therefore, the decision to take a developmental course involves a trade-off: an increased 
chance of eventually succeeding in the standard course, versus extra time and money. 

I want to find out how you see these trade-offs. 
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Form D 



Students' Satisfaction With Grades 
and Course Placement Decisions 



A research project by Richard Sawyer, ACT, 
with the cooperaiiun of Midwest Public University 

Fall, 1994 



Purpose of this Study 

There are benefits, risks, and costs associated with the decision to go to college. Part of ACT's 
work involves helping students decide which courses to take. I want to learn about the things 
you think about in making decisions about your courses. 

I will ask you some questions about your academic work at Midwest Public University, and 
about your preferences for different grades and course placement decisions. This questionnaire 
is not a test — there are no right or wrong answers. I do not ask you for your name or other 
identifying information on the questionnaire, so your answers will be anonymous. I have 
written a number on the top of this page, but only to help me keep track of which 
questionnaires are given to which classes. 

Your instructor will distribute this questionnaire in class. Please take it home, answer the 
questions, and then bring it back to the next class meeting, where we will collect it. 

The questions are grouped into four parts. As soon as you finish one part, please continue on 
directly to the next part. The entire questionnaire should take about 15 minutes to complete. 

The information you give will be used to enhance the services ACT provides to students in the 
future. I sincerely appreciate your cooperation. 
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Part 1 

Background Information 



1. Please check (/) the appropriate boxes to indicate whether you have taken, or are currently 
enrolled in, any of the courses in the table below. Also indicate either the grade you 
received in the course (if you have already taken it), or the grade you expect to receive (if 
you are currently enrolled in the course). 



Course 


Check here if 

you have 
already taken 


Grade 
you 
received 


Check here if you 
are currently 
enrolled 


Grade you 
expect to 
receive 


Basic Algebra II 










Elementary Functions 










Quantitative Methods I 










Calculus I 










Other mathematics courses 











(please specify): 



2. What is your major at Midwest Public University? (If you have not yet selected a major, 
please write "None.") 



3. When did you first start taking courses at Midwest Public University? 
(month and year) 

ERIC 



4. 



When did you first select your current major at Midwest Public University? 
(month and year) 



2 



5, What is your gender? 

Female 

Male 



6. What is your age? 
years 



Which of the following statements best describes your goals about the grades you earn in 
courses at Midwest Public University? (Check one only.) 

I don't mind earning a few Ds, so long as I receive credit for all my courses. 

It is important for me to earn only As, Bs, or Cs in my courses. 

It is important for me to earn only As or Bs in my courses. 

It is important for me to earn all As in my courses. 
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Part 2 
Course Grades 

Students want to earn as high a grade in a course as they can. Naturally, everyone would be more 
satisfied with an A than with a B, or with a B than with a C, and so forth — but what about your 
relative satisfaction? Would you, for example, feel twice as satisfied with an A as with a B? 

I want to find out your relative satisfaction with grades in the courses you take. In answering the 
questions, please think of any course that you need to pass to satisfy the requirements of your major 
at Midwest PubHc University. 

The line below is meant to suggest your relative satisfaction with the different letter grades. The letter 
grade of F is associated with 0% satisfaction, and the letter grade of A is associated with 100% 
satisfaction: 



F A 

J \ \ ! \ I 1 I I I I 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 



Please indicate on this line your relative satisfaction with the grades of B, C, and D by writing them 
above an appropriate point on the Hne. For example, if you would be about half as satisfied with a 
B as with an A, then you would write a "B" above the 50% mark. 

NOTE: Your responses should reflect your satisfaction with particular grades in a standard course. 
Your responses do not have to correspond to a percent-correct grading scale (for example, where the 
grade A represents 90% or more correct). 
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Course Grades (cont'd) 
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Sometimes we are given a choice between receiving a certain prize for sure, or else taking a chance 
on winning a better prize. For example, a contestant on a television game show may be given the 
choice either of wiiuiing a fancy color television for sure, or else having a 50% chance at winning 
$2,000 in cash (and a 50% chance of winning nothing). If the contestant already has a color television, 
then he or she might be willing to take a 50% chance at winning $2,000 in cash. On the other hand, 
the contestant might choose the certain prize of the color television if he or she doesn't have one. 



The following ten questions ask about your satisfaction with different grades in this way. Each 
question asks you to think about either earning a certain grade for sure [Option (1)], or else taking a 
chance, in which you might earn a higher grade, but also might earn a lower grade [Option (2)]. 

In answering these questions, please think of any course that you need to pass to satisfy the 
requirements of your major at Midwest Public University. 



1. Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either a C or an F. 

How large would the chances of earning a C have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of D [Option 

I would want the chances of earning a C to be % before taking a chance. 



2. Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either a B or an F. 

How large would the chances of earning a B have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of D [Option (1)]? 

I would want the chances of earning a B to be % before taking a chance. 



id i)l 



Suppose I offered you the choice of: 

(1) Earning exactly a grade of D for sure, or 

(2) Taking a chance, where you could earn either an A or an F. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of D [Option (1)]7 

I would want the chances of earning an A to be % before taking a chance. 



Suppose I offered you the choice of: 

(1) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either a B or an F. 

How large would the chances of earning a B have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of C [Option (1)]7 

I would want the chances of earning a B to be % before taking a chance. 



Suppose I offered you the choice of: 

(J) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either a B or a D. 

How large would the chances of earning a B have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of C [Option 

I would want the chances of earning a B to be % before taking a chance. 



Suppose I offered you the choice of: 

(1) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either an A or an F. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of C [Option 

I would want the chances of earning an A to be % before taking a chance. 
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Suppose 1 offered you the choice of: 

(1) Earning exactly a grade of C for sure, or 

(2) Taking a chance, where you could earn either an A or a D. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)1 to the sure grade of C [Option (1 )]? 



I would want the chances of earning an A to be 



_% before taking a chance 



8. Suppose I offered you the choice of: 

(1) Earning exactly a grade of B for sure, or 

(2) Taking a chance, where you could earn either an A or an F. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)] to the sure grade of B [Option (V]7 



I would want the chances of earning an A to be 



% before taking a chance* 



Suppose I offered you the choice of: 

(1) Earning exactly a grade of B for sure, or 

(2) Taking a chance, where you could earn either an A or a D. 

How large would the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)1 to the sure grade of B [Option (2)1? 



I would want the chances of earning an A to be 



% before taking a chance* 



10. Suppose 1 offered you the choice of: 

(2) Earning exactly a grade of B for sure, or 

(2) Taking a chance, where you could earn either an A or a C. 

How large w(^uld the chances of earning an A have to be before you would prefer taking a chance 
[Option (2)1 to the sure grade of B [Option (2)1? 



I would want the chances of earning an A to be 



% before taking a chance. 
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Part 4 
Course Placement 



Let a "standard course" be a for-credit course that is required for your major. For 
example, some students may need to pass Quantitative Methods I to satisfy the 
requirements of their major at Midwest Public University. 



One purpose of a course placement system is to determine whether a student is ready 
to take a particular standard course. If a student is not ready to take the standard 
course,- he or she can instead enroll in a "developmental course" to acquire the skills 
needed to succeed in the standard course. At Midwest Public University, for example, 
Basic Algebra II would be considered a developmental course for the standard course 
Quantitative Methods I. 



Taking a developmental course will tend to increase a student's chances of success in the 
standard course. However, taking a developmental course also has disadvantages — it 
increases the time required to complete your program, it costs additional money, and 
it may not carry credit toward your degree. Therefore, the decision to take a 
developmental course involves a trade-off: an increased chance of eventually succeeding 
in the standard course, versus extra time and money. 



I want to find out how you see these trade-offs. 



0 4 



00 



a 
c 

:2 .£ 

iS 

^ 2 

^ 6 
0^ 9) 

I 8 
as 

I' 

bO > 
I- 

o *^ 
c 

> c/i 

> {fi 

O < 

c S 



Of 
O 

u 



o c 



c 



c ^ 
a . 

|i 

8 

*^ *^ 

o c w: 

r- re C 



u 



o 



c 



o 
u 

03 



o 
U 
c 

X 



o 
U 
c 

X 

x: 



x: 
u 



c 

o 



u 



o 



o 
5 



o 



2 

o 
w 

C 

-S 2 

^ 6 
^ re 

o ^ 



u 



u 



o 
U 



5! 
-A 

e 

re 3 
c 8 

E ^ 

O T3 



3 
o 



re 

c 

rs 



0^ 

C 

T3 
re 



c 

CD 

H 



0^ 

o 



3 
o 



a; 
C 

.2 

a; 
a; 



c 
a; 

a; 

C/5 



* 



c 
o 

re 

s 

SB 



ERJC 



Form E 



Mathematics Instructors' Satisfaction 
With Grades and Course Placement Decisions 



A research project by Richard Sawyer, ACT 

Fall, 1994 



Purpose of this Study 

There are benefits, risks, and costs associated with the decision to take any college 
course. Part of ACT's work involves helping students decide which courses to take. I 
want to investigate mathematics instructors' satisfaction with the results of course 
placement decisions. 

I will ask you some questions about your teaching responsibilities at Midwest Public 
University and about your satisfaction with different levels of student performance. This 
questionnaire is not a test — there are no right or wrong answers. Because I do not ask 
you for your name or other identifying information on the questionnaire, your answers 
will be anonymous. 

The questions are grouped into four parts. As soon as you finish one part, please 
continue on directly to the next part. The entire questionnaire should take about 15 
minutes to complete. I will collect your completed questionnaire, along with those of 
your students at your next class meeting. 

The information you give will be used to enhance the services ACT provides to faculty 
and students in the future. I sincerely appreciate your cooperation. 
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Part 1 

Background Information 



1. Please check (/) the appropriate boxes to indicate whether you are currently teaching, or 
have taught, any of the courses in the table below. If you have taught the course, please 
estimate roughly the percentages of different grades your students earned. Naturally, 
grades vary from term to term, depending on your students' performance; please try to 
approximate what the typical percentages are. 



Course 


Check here if 
you are now 
teaching 


Check here 
if you have 
taught 


Approximate percentage 
of students who earned . . . 


A 


B 


C 


D 


F 


Basic Algebra 11 
















Elementary Functions 
















Quantitative Methods I 
















Calculus I 

















2. When did you first start teaching ' ourses at Midwest Public University? 
(year) 



2 



3. Please check (✓) the appropriate boxes to indicate how important the following factors are 
to you in awarding grades. 



Factor 


Importance in awarding grades 


Very 
important 


Important 


Somewhat 
important 


Not 
important 


Academic performance (as measured 
by tests, essays, homework, etc.) 










Attendance and participation in class 










Motivation and effort 










Other characteristics 
(please describe): 











4. Which of the following statements best describes your policy in assigning grades? 
(Check one only,) 

I grade strictly according to fixed standards of student performance. Therefore, I 

could (at least in principle) assign all As or all Fs. 

I grade strictly "on a curve": I always assign a certain percentag ? of As, a certain 

percentage of Bs, etc. 

I grade mostly according to a fixed standard, but I may modify some grades so 

that the distribution of grades meets a target grade distribution. 

I grade mostly "on a curve," but I may modify some grades if students' 

performance merits doing so. 



3 



Part 2 
Course Grades 



I want to find out your relative satisfaction with different levels of student perfonnance in the courses 
you teach. In responding to the questions, please think of the course you now teach that is often 
taken by first-year students. 

Instructors award grades on the basis of their students' academic achievement and other performance 
characteristics. The line below is meant to represent your satisfaction with student performance 
characteristics that would result in your assigning different letter grades. To simplify the discussion, 
I have associated F-level performance with 0% satisfaction, and A-level performance with 100% 
satisfaction: 



F-level 
perf* 



A-level 
perf. 



09c 



10% 



20% 



30% 



40% 



50% 



60% 



70% 



SOVv 



90% 



100% 



Please indicate on this line your relative satisfaction with B-leveK C-level, and D-level perfonnance by marking 
the letters "B", "C", and "D" at appropriate points above the line. For example, if you feel about half as 
satisfied with the performance of a student who earns a B as you do with the performance of a student who 
earns an A, then you would write a "B" above the 50% mark. 

Please note that your responses should reflect your satisfaction with particular levels of student performance. 
Your responses need not correspond to a percent-correct grading scale (where, for example, an A represents 90% 
or more correct). 
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Part 3 

Course Grades (cont'd) 
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The following questions are also related to your satisfaction with different levels of student 
performance. When responding to the questions, please think of the course you now teach that is 
often taken by first-year students. 

In each question, you are asked to choose between two hypothetical scenarios involving student 
performance in your course: In Scenario (1), all students perform at exactly the same level (e.g., every 
student performs at a level to which you would assign a grade of D). In Scenario (2), a certain 
percentage of students perform at a higher level, and all the rest perform at lower level (e.g, 75% 
perform at C level, and 25% perform at F level). Neither of these scenarios is realistic; but, by 
comparing your responses to different questions, I can estimate your relative satisfaction with different 
levels of student performance. 



1. Suppose 1 asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at D level, or 

(2) A mixed result, in which P percent of the students perform at C level, and all the rest perform at F level. 

How large would the percentage P of students who perform at C level have to be before you would prefer the mixed 
rc?sult (Scenario (2^1 to the uniform performance at D level [Scenario (7^1? 

I would want P = before selecting the mixed result. 



2. Suppose 1 asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at D level, or 

(2) A mixed result, in which P percent of the students perform at B level, and all the rest perform at F level. 

How large would the percentage P of students who perform at B level have to be before you would prefer the mixed 
result IScenario (2)\ to the uniform performance at D level IScenario 0^1? 



3. Suppose I asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at D level, or 

(2) A mixed result, in which P percent of the students perform at A level, and all the rest perform at F level. 



I would want P = 



before selecting the mixed result. 



How large would the percentage P of students who perform at A level have to be before you would prefer the mixed 
result IScenario (2)] to the uniform performance at D level IScenario 



I would want P = before selecting the mixed result. 
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4. Suppose J asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at C level, or 

(2) A mixed result, in which P percent of the students perform at B level and all the rest perform at F level. 

How large would the percentage P of students who perform at B level have to be before you would prefer the mixed 
result [Scenario (2)] to the uniform performance at C level [Scenario (1)]? 

I would want P = before selecting the mixed result 



5. Suppose I asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at C level, or 

(2) A mixed result, in which P percent of the students perform at B level, and all the rest perform at D level. 

How large would the percentage P of students who perform at B level have to be before you would prefer the mixed 
result [Scenario (2)] to the uniform performance at C level [Scenario fl)]? 

I would want P = before selecting the mixed result 



6. Suppose I asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at C level, or 

(2) A mixed result, in which P percent of the students perform at A level, and all the rest perform at F level. 

How large would the percentage P of students who perform at A level have to be before you would prefer the mixed 
result [Scenario (2)] to the uniform performance at C level [Scenario (1)]? 

I would want P = before selecting the mixed result. 



7. Suppose I asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at C level, or 

(2) A mixed result, in which P percent of the students perform at A level, and all the rest perform at D level. 

How large would the percentage P of students who perform at A level have to be before you would prefer the mixed 
result (Scenario (2)\ to the uniform performance at C level [Scenario (1)]? 

I would want P = before selecting the mixed result. 
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8. Suppose 1 asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at B level, or 

(2) A mixed result, in which P percent of the students perform at A level, and all the rest perform at F level. 

How large would the percentage P of students who perform at A level have to be before you would prefer the mixed 
result IScenario (2)] to the uniform performance at B level [Scenario (J)]7 



9. Suppose I asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at B level, or 

(2) A mixed result, in which P percent of the students perform at A level, and all the rest perform at D level. 

How large would the percentage F of students who perform at A level have to be before you would prefer the mixed 
result [Scenario (2)] to the uniform performance at B level (Scenario (J)]? 



10. Suppose I asked you to choose between the following two scenarios in your course: 

(1) A uniform result, in which all students perform at B level, or 

(2) A mixed result, in which P percent of the students perform at A level, and all the rest perform at C level. 

How large would the percentage P of students who perform at A level have to be before you would prefer the mixed 
result [Scenario (2)] to the uniform performance at B level [Scenario 



I would want P = 



before selecting the mixed result 



I would want P = 



before selecting the mixed result. 



I would want P 



before selecting the mixed result. 
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Background information . 

Let a "standard course" be a for-credit course that is taken by well-prepared entering 
students, and that is required for a major. For example, well-prepared entering students 
may take Quantitative Methods I and may need to pass it to satisfy the requirements of 
some majors at Midwest Public University. 



One purpose of a course placement system is to determine whether a student is ready 
to take a particular standard course. If a student is not ready to take the standard 
course, he or she can instead enroll in a "developmental course" to acquire the skills 
needed to succeed in the standard course. At Midwest Public University, for example, 
Basic Algebra II would be considered a developmental course for the standard course' 
Quantitative Methods I. 



Taking a developmental course will tend to increase a student's chances of success in the 
standard course. However, taking a developmental course also has disadvantages: It 
increases the time required to complete a program, it costs additional money, and it may 
not carry credit toward a degree. Some students may be discouraged from even starting 
a program if they have to take developmental courses. Therefore, the decision to take 
a developmental course involves a trade-off: an increased chance of eventually 
succeeding in the standard course versus extra time and money. 



I want to find out how you see these trade-offs. 
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Appendix B 



Eliciting von Neumann-Morgenstem Utilities 
for Coxirse Grades 



Table 1. Comparisons of Sure Events and Lotteries, and Their 
Associated Linear Equations 

Table 2. Systems of Full-Rank Linear Equations 
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Table 1. 

Comparisons of Sure Events and Lotteries, 
and Their Associated Linear Equations 





Comparison 




Linear Equation 


1 


B ~ lA, p„ Fl 


1 


w(B) = p, 


2 


B~\A,p„ D] 


2 


M(B) = p, + (I-Pj)iHD) 


3 


B ~ [A, p„ CI 


3 


u(B) = p3 + (l-p3)u(C) 


4 


C ~ [A, p„ Fl 


4 


K(C) = ^4 


5 


C ~ |A, p„ D] 


5 


U(C) = Pr, + (l-p,) «(D) 


6 


C ~ IB, p„ Fl 


6 


h(C) = p, im 


7 


C ~ IB, p„ Dl 


7 


n{C)=p:im + O-p.MD) 


8 


D~\A,p„ Fl 


8 


m(D) = 


9 


D ~ IB, p„ Fl 


9 


u(D) = w(B) 


K) 


D ~ |C, p,„ Fl 


10 


m(D) = Pi, m(C) 
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Table 2. 

Systems of Full-Rank Linear Equations 



System 


Equations* 


System 


Equations* 


System 


Equations* 


1 


I, 2, 3 


26 


1, 7,8 


51 


2, 6, 9 


2 


I, 2, 4 


27 


1, 7, 9 


52 


2, 6, 10 


3 


1, 2, 5 


28 


1, 7, 10 


53 


2, 7, 8 


4 


1, 2, 6 


29 


1, 8, 10 


54 


2, 7, 10 


5 


1, 2, 7 


30 


1, 9, 10 


55 


2, 8, 10 


6 


1,2, 10 


31 


2, 3,4 


56 


2, 9, 10 


7 


1, 3, 5 


32 


2, 3, 5 


57 


3, 4, 5 


8 


I, 3, 7 


33 


2, 3, 6 


58 


3, 4, 7 


9 


1, 3, 8 


34 


2, 3,7 


59 


3, 4, 8 


10 


1, 3, 9 


35 


2, 3,8 


60 


3, 4, 9 


11 


1, 3, 10 


36 


2, 3,9 


61 


3, 4, 10 


12 


1, 4, 5 


37 


2, 3, 10 


62 


3, 5, 6 


13 


1, 4,7 


38 


2, 4,5 


63 


3, 5, 7 


14 


1, 4, 8 


39 


2, 4,6 


64 


3, 5, 8 


15 


1, 4, 9 


40 


2, 4,7 


65 


3, 5, 9 


16 


1.4, 10 


41 


2, 4,8 


66 


3, 5, 10 


17 


1, 5, 6 


42 


2, 4,9 


67 


3, 6, 7 


18 


h 5, 7 


43 


2, 4, 10 


68 


3, 6, 8 


19 


1, 5, 8 


44 


2, 5,6 


69 


3, 6, 9 


20 


1, 5 9 


45 


2, 5,7 


70 


3, 6, 10 


21 


1, 5, 10 


46 


2, 5,8 


71 


3, 7,8 


22 


1, 6, 7 


47 


2, 5,9 


72 


3, 7,9 


23 


1, 6, 8 


48 


2, 5, 10 


73 


3, 7, 10 


24 


1, 6, 9 


49 


2, 6,7 


74 


3, 8,9 


25 


1, 6, 10 


50 


2, 6,8 


75 


3, 8, 10 



Equations enumerated in Table 1. 
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Table 2 (cont'd.) 
Systems of Full-Rank Linear Equations 



System 


Equations* 


System 


Equations* 


76 


3, 9, 10 


101 


6, 7, 10 


77 


4, 5, 6 


102 


6, 8,9 


78 


4, 5,7 


103 


6, 8, 10 


79 


4, 5,9 


104 


6, 9, 10 


80 


4, 5, 10 


105 


7, 8,9 


81 


4, 6,7 


106 


7, 8, 10 


82 


4, 6,8 


107 


7, 9, 10 


83 


4, 6,9 


108 


8, 9, 10 


84 


4, 6, 10 






85 


4, 7, 8 






86 


4,7,9 






87 


4, 7, 10 






88 


4, 8,9 






89 


4, 9, 10 






90 


5, 6,7 






91 


5, 6, 8 






92 


5, 6,9 






93 


5, 6, 10 






94 


5, 7,8 






95 


5, 7,9 






96 


5, 7, 10 






97 


5, 8,9 






98 


5, 9, 10 






99 


6, 7,8 






100 


6, 7, 9 







* Equations enumerated in Table 1. 
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Appendix C 



Constructing Summary Value Functions 
for Course Placement Outcomes 



Table 1. Coherent Choice Sequences 

Table 2. Imputed Summary Value Functions for Course Placement 
Outcomes 
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Constructing Summary Value Functions for Course Placement Outcomes 



Part 4 of each questionnaire asks respondents to make 10 choices. Each choice involves either 
taking the remedial course before taking the standard course, and earning grade Gi in the standard course, 
or else taking the standard course directly, and earning grade G2, where > G2. 

The result of the choices is a sequence of Rs and Ss, where: 

R= Prefer to take the remedial course before taking the standard course. 
S= Prefer to take the standard course directly. 

There are 2^^ = 1024 possible sequences of resjx)nse patterns, but most of them arc "incoherent," because 

they are inconsistent with the transitivity property of preference relations. A coherent sequence is one 

that satisfies the following inequalities: 

a. (R,A) y (R,B) > (R,C) > (R,D) > (R,F), and 

b. (S,A) > (S,B) > (S,C) y (S,D) > (S,F), 

where > is a respondent's preference. Then Inequalits^ a. implies, for example, that if (S,C) > (R,A), then 
(S,C) y (R,B), because (R,A) > (R,B). Moreover, Inequality b. implies that if (S,C) > (R,A), then 
{S,B) > (R,A), because (S,B) > (S,C). To simplify matters, I have also assumed that the following 
preferences exist: 

c. (R,A) > (S,F) 
(R,B) . (S,F) 
(R,C) > (S,F) 
(R,D) > (S,F) 

The inequalities in c. imply that in Choices 4, 7, 9, and 10, the respondent must always choose taking the 
remedial course and earning a passing grade in preference to taking the standard course directly and 
receiving an F. These preferences may not actually be true of students who are very willing to take risks. 
Making these assumptions, however, considerably reduces the number of allowable sequences. Finally, 
I assume that: 

d. (S,F) y (R,F) 

Inequalities a. - d. imply that every other course placement result is preferable to (R,F) (i.e., taking the 
remedial course and then receiving an F in the standard course). Table 1 on p. 3 shows the 14 choice 
sequences that satisfy these inequalities. 1 computed for each respondent an indicator function CHCSEQ: 

ERLC 



Appendix C 
Page 2 

CHCSEQ=1, if the respondent's sequence of choices was one of those listed in Table 1; and CHCSEQ=0, 
otherwise. The respondents for whom a coherent summary value function could be imputed were those 
for whom both CHR1ND1=1 (where CHRIND, is the coherence indicator for the grade value function) 
and CHCSEQ=1. These people were identified by the summary value function coherence indicator 
CHR1ND2=CHRJND1 *CHCSEQ. 
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If we use the customary grades A-F to measure achievement in the standard course, and if we 
neglect Withdrawal (W) grades, then there are 10 possible final outcomes of the placement system: 
X = {(R,A), . . (R,F), (S,AX . . V (S,F)}, where R denotes taking the remedial course before taking the 
standard course, and S denotes taking the standard course directly. The set X, together with the set of 
possible placement test scores is the outcome space 0. 

in principle, one could elicit a value function for X with a diagram like that in Part 2 of the 
questionnaires. With such a diagram, however, the respondent would have to mark 8 outcomes (rather 
than the 3 outcomes A, B, and C) above the 0-100 scale. I believe that most respondents would have great 
difficulty doing this. Therefore, I elected to iinpute a summary value function svf for X, using the grade 
value function ^/elidied in Part 2 of the questionnaires as a reference. Now, there are many ways one 
could impute a summary value function; I chose the simplest method I could think of. Specifically, the 
imputed value function svf has the following properties: 

a. stf(5,G) = gvfiC), for G = A, B, C, D 

b. svf(R,F) = 0. 

c. 5U/(S,F) = .25*^^/(0) 

d. For G = A, B, C, D, the values of stf(R,G) are interpolated between appropriate values of 
1=X^/(A), gvm gvfiCX ^f{Dl and 0. 

Equation a. says that the summary value function associated with taking the standard course directly and 

earning a particular grade G is equal to the grade value function gvf elicited in Part 2 of the 

questionnaires. Equation b. says that the worst possible result is to take the remedial course, then receive 

an F in the standard course. Equation c. says that taking the standard course directly, and receiving an 

F is slightly better than receiving an F in the standard course after taking the remedial course; I have 

arbitrarily assigned the value .25*^(0) to this result. Property d. says that the outcomes associated with 

first taking the remedial course are to be assigned values according to the respondent's 10 choices in Part 

4 of the questionnaires. Provided that the respondent's sequence of choices is one of the 14 coherent 

sequences listed in Table 1, it is possible to interpolate between values of gvf in a consistent way. Each 

of the 14 coherent choice sequences defines a separate imputed summary value function svf. The resulting 

values of the imputed summary value functions svf are shown in Table 2 on the following page. 
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